Credit: The KonG / Shutterstock
For all of its imperfections, generative artificial intelligence can perform tasks associated with more than 80% of jobs in the U.S., according to one estimate, with highly skilled jobs and occupations requiring advanced degrees particularly affected. But the degree to which using these tools truly helps organizations and their employees remains largely unknown.
“It’s a challenge to measure how AI is affecting productivity in a natural workplace environment,” said an assistant professor of economics at MIT Sloan. “You can always run a lab experiment where you give a task to workers and ask one group to do it using AI, but that, of course, is an artificial setting.”
In a new research paper, Demirer and his co-authors analyzed the rollout of an AI coding assistant at three technology companies. The researchers found that introducing a generative AI tool to software developers did increase productivity, with less-experienced developers showing higher adoption rates and greater productivity gains.
The paper was coauthored by Zheyuan (Kevin) Cui of Princeton, Leon Musolff of the University of Pennsylvania, Sonia Jaffe and Sida Peng of Microsoft, and Tobias Salz of MIT.
Using AI for software development
The researchers worked with Microsoft, Accenture, and an anonymous Fortune 100 electronics manufacturing company, each of which was running its own experiment with GitHub Copilot, an AI-based coding assistant that suggests intelligent code completions. A subset of software engineers was able to use the tool before all developers had access. (Demirer emphasized that the companies designed and ran the experiments; he and his colleagues analyzed the data.)
Related Articles
“Software development, it turns out, is a very good setting for looking at the value of AI,” Demirer said. “If you think about [AI’s] benefit to managers or salespeople, it’s a lot harder to pin down their output and measure it day to day. Software developers work in an environment where most of their output is measurable and clearly recorded.”
In the Microsoft and Accenture experiments, one group of developers was randomly given access to Copilot and a second group did not have access to the tool. At Microsoft, this split lasted for seven months; at Accenture, it lasted for four months. At the anonymous company, all users were given access to Copilot over a period of two months, but this happened in a staggered fashion, with some teams using the tool up to six weeks before others.
AI helped newer employees with less experience
The researchers found that access to Copilot increased output — the number of completed weekly tasks — by 26% when the results across all three experiments were averaged. However, the biggest productivity gains were seen among recent hires and developers in more junior positions, who increased their output by 27% to 39%. More senior developers saw productivity gains of 8% to 13%.
“Inexperienced and short-tenured software developers were more likely to use the tool, and, moreover, their productivity increased a lot more,” Demirer said. “For those who are more experienced we actually don’t see much of an effect.”
This result could be related, in part, to the fact that newer and lower-ranking developers adopted Copilot at higher rates, Demirer said. He also made clear that he and his colleagues did not have access to the code that was produced, which means that although they were able to measure productivity, they were unable to evaluate the quality of the work produced with Copilot. Taking that second measure is an important next step, he said.
Questions remain about future AI adoption
One of the ancillary findings that is quite significant, Demirer said, is that the average rate of AI adoption at the three companies was only about 60% after one year. Even though Copilot is easy to use, companies trying to deploy such tools may need to account for a long road to full adoption; in this regard, generative AI is no different from other new technologies.
Leading the AI-Driven Organization
In person at MIT Sloan
Register Now
A more fundamental question is the extent to which AI, given its incredibly rapid advancement, will serve as a substitute for or a complement to highly skilled workers. One potential future with generative AI has companies doing the same amount of work with half as many workers; another potential future, represented by this experiment, has employees getting their work done more quickly and thus having more time to work on other projects.
“From this case study, we know that productivity effects are real, and they will lead to important changes in the workplace,” Demirer said. “But what, exactly, those will look like five or 10 years from now we don’t know. That’s the big question.”
Read the paper: The Effects of Generative AI on High Skilled Work
